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Abstract 

We develop several lower bounds on the capacity of binary input symmetric output channels with synchronization 
errors which also suffer from other types of impairments such as substitutions, erasures, additive white Gaussian 



C/3 ' noise (AWGN) etc. More precisely, we show that if the channel with synchronization errors can be decomposed 

O 

into a cascade of two channels where only the first one suffers from synchronization errors and the second one 

. is a memoryless channel, a lower bound on the capacity of the original channel in terms of the capacity of the 

^O ' synchronization error-only channel can be derived. To accomplish this, we derive lower bounds on the mutual 

CO , information rate between the transmitted and received sequences (for the original channel) for an arbitrary input 



distribution, and then relate this result to the channel capacity. The results apply without the knowledge of the exact 
capacity achieving input distributions. A primary application of our results is that we can employ any lower bound 
derived on the capacity of the first channel (synchronization error channel in the decomposition) to find lower bounds 
on the capacity of the (original) noisy channel with synchronization errors. We apply the general ideas to several 

X' 

^ . specific classes of channels such as synchronization error channels with erasures and substitutions, with symmetric 

g-ary outputs and with AWGN explicitly, and obtain easy-to-compute bounds. We illustrate that, with our approach, 
it is possible to derive tighter capacity lower bounds compared to the currently available bounds in the literature 
for certain classes of channels, e.g., deletion/substitution channels and deletion/ AWGN channels (for certain signal 
to noise ratio (SNR) ranges). 

Index Terms 
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I. Introduction 

Depending on the transmitting medium and the particular design, different Umiting factors degrade the perfor- 
mance of a general communication system. For instance, imperfect alignment of the transmitter and receiver clocks 
may be one such factor resulting in a synchronization error channel modeled typically through insertion and/or 
deletion of symbols. Other factors include the effects of additive noise at the receiver etc. The main objective of 
this paper is to study the combined effects of the synchronization errors and additive noise type impairments and in 
particular to "decouple" the effects of the synchronization errors from the other parameters and obtain expressions 
relating the channel capacity of the combined model and the synchronization error-only channel. 

We focus on achievable rates for channels which can be considered as a concatenation of two independent channels 
where the first one is a binary channel suffering only from synchronization errors and the second one is either a 
memoryless binary input symmetric q-ary output channel (BSQC) or a binary input AWGN (BI-AWGN) channel. 
For instance, the first channel can be a binary insertion/deletion channel and the second one can be a binary 
symmetric channel (BSC) or a substitution/erasure channel (a ternary output channel q = 3). Our development 
starts with the ternary {q = 3) and quaternary {q = 4) output cases, respectively, then we generalize the results to 
a general q-aiy output case. Specifically, we obtain achievable rates for the concatenated channel in terms of the 
capacity of the synchronization error channel by lower bounding the information rate of the concatenated channel for 
input distributions which achieve the capacity of the synchronization error-only channel and the parameters of the 
memoryless channel. The lower bounds are derived without the use of the exact capacity achieving input distribution 
of the synchronization error channel, hence any existing lower bound on the capacity (of the synchronization error- 
only channel) can be employed to obtain an achievable rate characterization for the original channel model of 
interest. 

By channels with synchronization errors we refer to the binary memoryless channels with synchronization errors 
as described by Dobrushin in yj] where every transmitted bit is independently replaced with a random number 
of symbols (possibly empty string, i.e. a deletion event is also allowed), and the transmitter and receiver have no 
information about the position and/or the pattern of the insertions/deletions. Different specific models on channels 

with synchronization errors are considered in the literature. Insertion/deletion channels are used as common models 

n r 

for channels with synchronization errors, e.g., the Gallager insertion/deletion channel [2J, the sticky channel 131] 
and the segmented insertion/deletion channel ||4|]. 

Dobrushin |[l|] proved that Shannon's theorem applies for a memoryless channel with synchronization errors by 
demonstrating that information stability holds for memoryless channels with synchronization errors. That is, for the 
capacity of the synchronization error channel, Cs we can write Cs = lim max -—I{X; Y), where X and Y are 

N-^oo p(X) ^ 

the transmitted and received sequences, respectively, and N is the length of the transmitted sequence. Therefore, 



the infonnation and transmission capacities of the memoryless channels with synchronization errors are equal and 
we can employ any lower bound on the information capacity as a lower bound on the transmission capacity of a 
channel with synchronization errors. 

There are many papers deriving upper and/or lower bounds on the capacity of the insertion/deletion channels, 



e.g., see 115, 
errors, e.g. ||2, 




1011 : however, only a very few results exist for insertion/deletion channels with substitution 



nil or in the presence of AWGN, e.g. 012 



13ll . Our interest is on the latter, in fact, on more general 



models incorporating erasures as well as q-ary channel outputs. 

Let us review some of the existing relevant results on insertion/deletion channels in a bit more detail. In ||2|], 
Gallager considers a channel model with substitution and insertion/deletion eixors (sub/ins/del) where each bit gets 
deleted with probability p^, replaced by two random bits with probability pi, correctly received with probability 
Pc = (1 — Pd — Pi)(l ~ Ps), and changed with probability pj = (1 — p^ — Pi)ps, and derives a lower bound on the 
channel capacity (in bits/use) given by 



C >l+Pdlogpd+ Pi logpi + Pc logpc + Pf log(p/), 



(1) 



where log(.) denotes logarithm in base 2. Fertonani and Duman in 111 ill develop several upper and lower bounds 
on the capacity of the ins/del/sub channel, where they employ a genie-aided decoder that is supplied with side 
information about some suitably selected random processes. Therefore, an auxiliary memoryless channel is obtained 
in such a way that the Blahut-Arimoto algorithm (BAA) can be employed to obtain upper bounds on the capacity 
of the original channel. Furthermore, it is shown that by subtracting some quantity from the derived upper bounds 
which is, roughly speaking, more than extra information provided by the side information, lower bounds on the 



capacity can also be derived. In lll3ll . Monte Carlo simulation based results are used to estimate information rates of 
different insertion and/or deletion channels in the absence or presence of intersymbol interference (ISI) in addition 
to AWGN with independent uniformly distributed (i.u.d.) input sequences. In [12^, the synchronization errors are 
modeled as a Markov process and simulation results are used to compute achievable information rates of an ISI 



channel with synchronization errors in the presence of AWGN. In jlOll . Rahmati and Duman compute analytical 
lower bounds on the capacity of the i.i.d. del/sub and i.i.d. del/ AWGN channels, by lower bounding the mutual 
information rate between the transmitted and received sequences for i.u.d. input sequences focusing on small deletion 
probabilities. 

The paper is organized as follows. In Section JIJ we formally give the models for binary input symmetric q- 
ary output channels with synchronization errors and BI-AWGN channels with synchronization errors. In |III] we 
give two lemmas and one proposition which will be useful in the proof of the result on BSQC channels with 
synchronization errors. In Section |IV] we initially focus on a substitution/erasure/synchronization error channel 
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Fig. 1. Binary input symmetric g-ary output channel with synchronization errors. 

(abbreviated as sub/ers/synch channel) which is a binary input symmetric ternary output channel, and then on a 
binary input symmetric quaternary output channel. After that we extend the results to the case of more general 
symmetric q-ary output channels. In Section |Vl we lower bound the capacity of a synchronization error channel 
with AWGN (abbreviated as AWGN/synch channel) in terms of the capacity of the underlying synchronization 
error only channel. More precisely, we generalize the results on BSQC channels with synchronization errors when 
q goes to infinity. We present several numerical examples illustrating the derived results in Section |Vl] Finally, we 
conclude the paper in Section IVIII 

II. Channel Models 

A general memoryless channel with synchronization errors OIQ is defined via a stochastic matrix {p{yi\xi),yi G 
y, Xi S X} where X is the input alphabet (e.g., for a binary input channel X = {0, 1}), and 3^ is (possibly empty) 
the set of output symbols, < p{yi\xi) < 1, and J2y eyP(yi\^) = 1. As a particular instance of this channel, if 
PiVi = ^\xi) = Pd (0 denoting the null string) and p{yi = Xj) = 1 — pi, we obtain an i.i.d. deletion channel. 

A. Binary Input Symmetric q-ary Output Channel with Synchronization Errors 

By a binary input symmetric g-ary output channel (BSQC) with synchronization errors, we refer to a channel 
which can be considered as a concatenation of two independent channels, depicted in Fig [H such that the first one 
is a channel with only synchronization errors with input sequence X and output sequence Y, and the second one is 
a BSQC with input sequence Y and output sequence Y^"^' , where by a symmetric channel we refer to the definition 
given in il4i p. 94]. In other words, a channel is symmetric if by dividing the columns of the transition matrix into 
sub-matrices, in each sub-matrix, each row is a permutation of any other row and each column is a permutation 
of any other column. For example, a channel with independent substitution, erasure and synchronization errors 
(sub/ers/synch channel) can be considered as a concatenation of a channel with only synchronization errors with 
input sequence X and output sequence Y and a substitution/erasure channel (binary input ternary output channel) 
with input sequence Y and output sequence Y^^' . In a substitution/erasure channel, each bit is independently flipped 
with probability ps or erased with probability p^, as illustrated in Fig. |2] (a). Another example is a binary input 
symmetric quaternary output channel with synchronization errors which can be decomposed into two independent 
channels such that the first one is a memoryless synchronization error channel and the second one is a memoryless 
binary input symmetric quaternary output channel shown in Fig. |2] (b). 




Fig. 2. (a) Input-output relation in the substitution/erasure channel (P{Y^ mO for all 1 < i < |y|). (b) Input-output relation in the binary 
input quaternary output channel {P{Y^ lY) for all 1 < i < \y\). 
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Fig. 3. AWGN channel with synchronization errors. 

B. BI-AWGN Channels with Synchronization Errors 

In a BI-AWGN channel with synchronization errors, bits are transmitted using binary phase shift keying (BPSK) 
and the received signal contains AWGN in addition to the synchronization errors. As illustrated in Fig. |3] this 
channel can be considered as a cascade of two independent channels where the first one is a synchronization error 
channel and the second one is a BI-AWGN channel. We use X to denote the input sequence to the first channel 
which is a BPSK modulated version of the binary input sequence X, i.e., Xi = 1 — 2Xi and Y to denote the 
output sequence of the first channel and input to the second one. Y is the output sequence of the second channel 
that is the noisy version of Y , i.e., 

where Zj's are i.i.d. zero mean Gaussian random variables with variance o"^, and Yf" and Yf are the i*^ received 
and transmitted bits of the second channel, respectively. 



C. Simple Example of a Synchronization Error Channel Decomposition into Two Independent Channels 

The procedure in finding the capacity bounds used in this paper can be employed for any channel which can be 
decomposed into two independent channels such that the first one is a memoryless synchronization error channel and 
the second one is a symmetric memoryless channel with no effect on the length of the input sequence. Therefore, if 
we can decompose a given synchronization error channel into two channels with described properties, we can derive 
lower bounds on the capacity of the synchronization error channel. The advantage of this decomposition is that 
decomposing the original synchronization error channel into a well characterized synchronization error channel and 
a memoryless channel could be done in such a way that lower bounding the capacity of the new synchronization 
error channel be simpler than the capacity analysis of the original synchronization error channel. In Table H] we 



TABLE I 

Transition probabilities of the hypothetical synchronization error channel. 
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TABLE II 

Transition probabilities of two independent channels giving rise to the synchronization error channel given in 

Table [H 
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provide an example of a hypothetical channel with synchronization errors that can be decomposed into a different 
synchronization error channel and a memoryless binary symmetric channel (BSC). In Table |lll the two channels 
used in the decomposition are given. 

III. Entropy Bounds for Binary Input ^-ary Output Channels with Synchronization Errors 

In the following two lemmas, we provide a lower bound on the output entropy and an upper bound on the 
conditional output entropy of the binary input g-ary output channel in terms of the the corresponding output 
entropies of the synchronization error channel, respectively. We then give a proposition that will be useful in the 
proof of the result on BSQC channels with synchronization errors (note that the following two lemmas hold for 
any binary input g-ary output channels with synchronization errors regardless of any symmetry). 

Lemma 1. In any binary input q-ary output channel with synchronization errors and for all non-negative integer 
values of q, we have 



^y'"' y,p{y)7^o 



(2) 



where M is the random variable denoting the length of the received sequence, Y denotes the output sequence 
of the synchronization error channel and the input sequence of the binary input q-ary output channel, and Y^'^' 
denotes the output sequence of the binary input q-ary output channel. 

Proof: By using two different expansions of H(Y^'^\ M), we have 



H{Y'^''\m) = H{Y^'i^) + H{M\Y^'^^] 
= H{Y^'i\M) + H{M). 



(3) 



Hence, we can write 

H(Y^'^^) = H{Y^'^'^\M)+H{M), (4) 

where we used the fact that by knowing Y^'^', random variable M is also known, i.e. H{M\Y^'^') = 0. By using 
the same approach for H{Y), we have 

H(Y) = H(Y\M)+H{M). (5) 

Finally, we can write 



H{Y^'i'^)-H{Y) = H{Y^^'^\M)-H{Y\M) 

= ^p{m)\H(Y^'^^M = m)- H{Y\M = m) 

m 

where p{m) = P{M = m). On the other hand, due to the definition of the entropy, we can write 

iJ(r('?)|M = m) - H{Y\M = m) = £;y(,){-log(p(l^('')))|M = m} - -Eyj- log(p(Y))|M = m} 



(6) 



E^^i,) { -log 



Y. Y. P(.y^''^\y,M = m)p{y\M = m) log (^^^ 



M = m> 

' " M = m) 



where E^{.} denotes the expected value with respect to the random variable Z. Now due to the fact that — log(x) 
is a convex function of x, we apply Jensen's inequality to write 

H(Y^'''^\M = m)-H(Y\M = m) > - log | V V p(y^^^\y,M = m)p(y\M = m f^^, \ ^^ ^ ""^ 

I ^-^ ^-^ p[y\M = m) 

Vy"" y,p{y)7^o ^^^' ^ 

= -logK] Y. P{y^''^y,M = m)p{y^'i^\M = m)\. (7) 

By substituting this result into ©, the proof follows. ■ 

Lemma 2. In any binary input q-ary output channel with synchronization errors and for any input distribution, we 
have 

if(r('?)|X) < H{Y\X) + E{M}H{Y^'^\Yj), (8) 

where Yj denotes the j-th output bit of the synchronization error channel and j-th input bit of the binary input 
q-ary output channel and Y.- denotes the output symbol of the binary input q-ary output channel corresponding 
to the input bit Yj. 



Proof: For the conditional output entropy, we can write 

H{Y^'i\Y\X) = H{Y^'i\x)+H{Y\Y^'i\x) 
= H{Y\X) + H{Y^'i\Y,X) 
= H{Y\X) + H{y'^'^^\Y), (9) 

where the last equality follows since X ^ Y ^ Y^*^' form a Markov chain. Therefore, 

H{Y^''^\X) = H{Y\X) + H{Y^'i\Y)-H{Y\X,Y^'i^) 

< H{Y\X) + H{Y^'i\Y). (10) 

On the other hand, by using the fact that by knowing Y , M is also known, we have 

H{Y^'''^\Y) = H{Y'^'i\M,Y). (11) 

Furthermore, since the second channel is memoryless, we obtain 

H{Y^'^\y,M) = ^p{m)H{Y^'''^\Y,M = m) 

m 

m 

= Em{M}H{y}''^\Yj), (12) 

which concludes the proof. ■ 

By combining the results of Lemmas [U and |2l we obtain 

I{X;Y'i) >I{X;Y)- Em I log Iy^ Y^ p{y^'^^\y,M)p{y^'i^\M)\ i - i?{M}/7(y/'^V.), (13) 

I Vyc' yMy)7^o J J 

which gives a lower bound on the mutual information between the transmitted and received sequences of the 
concatenated channel I{X; Y'^) in terms of the mutual information between the transmitted and received sequences 
of the synchronization error channel I{X;Y). 

Proposition 1. For any X, Y and Y^'^^ forming a Markov chain X ^ Y ^ Y^'^\ if 

I{X;Y'^''^)>I{X;Y)+A, 
where A is a constant, then the capacity of the channels X — )■ Y^'^' (C ^ viit) '^^d X ^Y (Cj^^y^ satisfy 



Proof: Using the input distribution which achieves the capacity of the channel X ^Y , P{X), we can write 

lim -I(X;1"(«)(X)) > \im-I{X]Y{X))+A 

n— >-oo n 71— >oo n 

= Cx^Y + ^- (15) 

Hence, for the capacity of the channel X —^ "K '■''■', we have 

Cy viM) = lim - max/(X;r('?)) 

^^^-^ n-!>oo n P(X) 



> lim -I{X;Y^'i\X)) 



n— >oo n 

> Cx^Y+A- (16) 



Due to the result in ( fT3] ) and the result of Proposition [T] the capacity of the concatenated channel can be lower 
bounded in terms of the capacity of the synchronization error channel and the parameters of the second (memoryless) 
channel. 

IV. Achievable Rates over Binary Input Symmetric q-ary Output Channels with 

Synchronization Errors 

In this section, we focus on BSQC channels with synchronization errors (as introduced in Section III-Al i and 
provide lower bounds on their capacity. We first develop the results for sub/ers/synch channel and binary input 
symmetric quaternary output channel, respectively. Then give the results for general (odd and even) q, respectively. 

A. Substitution/Erasure Channels with Synchronization Errors 

The following theorem gives a lower bound on the capacity of the sub/ers/synch channel with respect to the 
capacity of the synchronization error channel. In a sub/ers channel, every transmitted bit is either flipped with 
probability of ps, or erased with probability of p^ or received correctly with probability of 1 — ps — Pe independent 
of each other. 

Theorem 1. The capacity of the sub/ers/synch channel Cges can be lower bounded by 

Cses >Cs-r [H{ps,Pe, 1 - Ps - Pe) + log {{1 - Pef + 2pfj] , (17) 

where Cg denotes the capacity of the synchronization error channel, r = lini„_!.oo "^ — > "- ^^d m denote the 
length of the transmitted and received sequences, respectively. 



10 

Before giving the proof of Theorem \T\ we consider some special cases of this result. Since we have considered 
the general synchronization error channel model of Dobmshin llj], the lower bound (ITtI i holds for many different 
models on channels with synchronization errors. A popular model for channels with synchronization errors is the 
Gallager's ins/del modejj in which every transmitted bit is either deleted with probability of p^ or replaced with 
two random bits with probability of pi or received correctly with probability of I — p^ — Pi independent of each 
other while neither the transmitter nor the receiver have any information about the insertion and/or deletion errors. 
If we employ the Gallager's model in deriving the lower bounds, for the parameter r, we have 

r = lim 

n— >oo n 

= lim —nEUsjl] 

n-s>oo n •-'■"-' 

= l-Pd+Pi, (18) 

where | Sj \ denotes the length of the output sequence in one use of the ins/del channel, and the equality results 
since the channel is memoryless. By utilizing the result of (fTSl ) in dTV]) . we obtain the following two corollaries. 

Corollary 1. The capacity of the sub/ers/ins/del channel Cseid is lower bounded by 

Cseid > Cid - (l-pd + Pi) [H{p„pe,l-Ps-Pe) + log {{I + Pe? + 2pl)] , (19) 

where Cid denotes the capacity of an insertion/deletion channel with parameters pd and pi. 

Taking pe = in this channel model gives the ins/del/sub channel, hence we have the following corollary. 
Corollary 2. The capacity of the ins/del/sub channel Cids can be lower bounded by 

Qds > Cid -{l-Pd+ Pi)Hb{Ps), (20) 

To prove Theorem [T] we need the following two lemmas. In the first one we give a lower bound on the output 
entropy of the sub/ers/synch channel related to the output entropy of the insertion/deletion channel, while in the 
second one we give an upper bound on the conditional output entropy of the sub/ers/synch channel, related to the 
conditional output entropy of the insertion/deletion channel. 

Lemma 3. For a sub/ers/synch channel, for any input distribution, we have 

H{Y^^^) > H{Y) - E{M} log ((1 - p,f + 2pl) , (21) 

where Y denotes the output sequence of the synchronization error channel and input sequence of the substitu- 

In fact, Gallager's model in general refers to a channel with insertion, deletion and substitution errors, but with Gallager's ins/del model 
we refer to the case with ps = (i.e., substitution error probability being zero). 



11 

tion/erasure channel, and Y^^> denotes the output sequence of the substitution/erasure channel. 
Proof: Using the result of Lemma [T] we only need to obtain an upper bound on 

for all values of m. On the other hand for p{y^^^\y, M = m), we have 

m 

p(y(3)|y,M = m) = llp{Y}''^\Yi) 

i=l 
= ptd'{l-Ps-Per-''~'^ (22) 

where j'l denotes the number of transitions — )• — or 1 — )• — and J2 denotes the number of transitions — )■ 1 or 
1 -^ 0. E.g., p(011 - |0000) = p{0\0)p{l\0)p{l\0)p{-\0) = Pepl{l-Pe -Ps)- On the other hand, for a fixed output 
sequence y^^> of length m with ji erased symbols " — ", there are 2^^ ('^T-'^) possibilities among all Tn-tuples such 
that d{y^'^')e = ji, i.e., the number of erased symbols in y^^', and d{y,y^^>)s = J2, i-c, the number of positions 
in y and y^^'' in which Y s are the flipped versions of Yj, therefore we can write 

m-ji 



J2 p{y^'^\y,M = m) < ^2^^r .^')p^^p^^{l-Ps-Pe 



y,piy)7^o J2=o 

2VeHl-Pe)™-^^ (23) 



Note that in deriving the inequality in (jT]), the summation is taken over the values of y with p{y) ^ 0. However, 
in (l23l ) the summation is taken over all possible values of y of length m (over all m-tuples), i.e. p{y) = or 
p{y) 7^ 0, which results in the lower bound in (l23T l. Furthermore, by using the fact that the probability of having 
jl erasures in a sequence of length m is equal to ("^)y4^(l — pe)'^~^\ we obtain 



y^p(y(^)|M = m) Y^ p{y^^^\y,M = m) < "^ P{d{y^^^)e = ji\M = m)2^^pi^{l - p 

y(3) y,p{y)7^o y(3) 



er-'' 



lil-Pe? + 2pir. (24) 



By substituting this result into (|2]), we arrive at 

H{Y^^y)-H{Y) > -E{M}log{{l+pef + 2pl), (25) 

concluding the proof. ■ 

It is also worth noting that any capacity achieving input distribution over a discrete memoryless channel results in 



12 

Strictly positive output probabilities for possible output sequences of the channel ( II14I. p. 95]). Therefore, for special 
synchronization error channel models in which for any possible length of the output sequence m, all the ?n,-tuple 
output sequences are probable, e.g. i.i.d. deletion channel or i.i.d. random insertion channel, capacity achieving 
input distributions {p{x)) would result in strictly positive output probability distributions for all m-tuple output 
sequences, i.e. p{y'^) > for all y'^ of length m and all possible m. Hence, the bounds in (l23l) and (l24l) can be 
thought as equalities for these cases. 

Lemma 4. In any sub/ers/synch channel and for any input distribution, we have 

HiY'-^^\X)<H{Y\X) + E{M}H{pe,Ps,l-Pe-Ps). (26) 

Proof: Due to the result of Lemma |2] and the fact that in a substitution/erasure channel, regardless of the 
distribution of Yj, we can write 

i7(yf Vj) = H{p,,ps,l-Pe-Ps), (27) 

hence the proof follows. ■ 

We can now complete the proof of the main theorem. 

Proof of Theorem |7] : By substituting the results of Lemmas |3] and |4] into the definition of mutual information, 
for the same input distribution given to both synchronization error and sub/ers/synch channels, we obtain 

I{X; f(3)) > /(X; Y) - E{M} [H{ps,pe, 1-Ps-Pe)+ log ((1 + Pe? + 2pl)] . (28) 

By using the result of Proposition [T] the proof is completed. ■ 

B. Binary Input Symmetric Quaternary Output Channels with Synchronization Errors 

In this subsection, we consider a binary input symmetric quaternary output channel with synchronization errors 
as described in Section JIl 

Theorem 2. The capacity of the binary input symmetric quaternary output channel with synchronization errors 
Csq can be lower bounded by 

Csq >C,-r [H{pup2,P3,Pi) + log ((pi +P3f + (j)2+P4f)] , (29) 

where Cg denotes the capacity of the synchronization error only channel, and r is as defined in (|17| ). 

Note that, the presented lower bound is true for all memoryless synchronization error channel models. Therefore, 
similar to the sub/ers/synch channel we can specialize the results to the Gallager insertion/deletion channel as given 
in the following corollary. 
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Corollary 3. The capacity of binary input symmetric quaternary output channel with insertion/deletion errors 
(following Gallager's model) Cqid is lower bounded by 

Cqid > Cid - {l-Pd+Pi) [H{pi,P2,P3,Pa) + log {{pi+P3? + (P2 +PAf)] ■ (30) 

To prove Theorem |2] we need the two lemmas below where the first one gives a lower bound on the output 
entropy of the binary input quaternary output channel with synchronization errors related to the output entropy of 
the synchronization error channel, and the second one gives an upper bound on the conditional output entropy of 
the binary input quaternary output channel with synchronization errors, related to the conditional output entropy of 
the synchronization error channel. 

Lemma 5. In any binary input quaternary output channel with synchronization errors and for any input distribution, 
we have 

//(yW) > H{Y) -E{M}log {{pi+psf + {P2+P4f) , (31) 

where Y denotes the output sequence of the synchronization error channel and input sequence of the binary input 
quaternary output channel, and Y^ > denotes the output sequence of the binary input quaternary output channel 
corresponding to the input sequence Y. 

Proof: Similar to the proof of Lemma |3] we use the result of Lemma [T] by taking the summation over all 
possible sequences of length m, i.e., regardless of p{y) = or p{y) ^ 0, which results into a looser lower bound. 
On the other hand, for p(y(^)|y, M = m), we have 

m 

(4), 



p(y(^)|y,M = m) = JJ^^^/ 1^' 



j=i 



= P^ip{'p^ipT'''''''\ (32) 

where ji denotes the number of transitions — )• 0~ or 1 — )• 1^, j2 denotes the number of transitions — )• 
0+ or 1 — )• 1+, and J3 denotes the number of transitions — )■ 1^ or 1 — > 0^. E.g., p(0^ 1+0+1^ |0000) = 
p(0^|0)p(l+|0)p(0+|0)p(l~|0) = PIP2P3P4- Furthermore, for a fixed output sequence y'-^) of length m with 
j 0~ symbols, k 0+ symbols, / 1~ symbols and m — j — k — I 1+ symbols, there are (/ ) (j3 (i ) (™~"i~ ~ ) 
possibilities among all m-tuples (for y) such that d{y,y^'^^)o^o- = h, d{y,y^'^^)o^o+ = 12, d(y,y^^^)o^i- = h 
and (i(y,y(^))o--5>i+ = i^- By defining ■m^{y^^^) = #{t < m\yl G {0^,1^}}, i.e., the number of the times 
y^ = 0~ or yl = 1^, and m'^{y^'^>) = #{t < m\yl G {0+, 1^}}, i.e., the number of the times yj = 0"^ or 
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(4) I 

yl = 1+, we can wnte 

il=0 ^ ^^ i2=0 ^ ^'^ 13=0 ^ ■^'^ i4=0 \ 4 / 

By taking the summation over all possible output sequences of length m, and using the fact that the probability of 
having the output y^'^'^ with length m containing m^ 0^ or 1^ is {^-){pi +P3)'" {P2 +P4)™~™ , we obtain 

J]]p(y(^)|M = m) J]]p(y(^)|y,M = m) = J]]p(yW|M = m){pi + pzT' ^y'''\p2 +PAr^^y'''^ 

yW y yW 

= E [^^)ipi+psr'iP2+P4r-"''{pi+P3r'{P2+P4r-'^' 

m-=0 ^ ^ 

= ((Pl+P3)' + (P2+P4)')" (34) 

By substituting the result of (l34l) into the result of Lemma [H we obtain 

F(yW) > H{Y) - Em {M} log {{pi+P3f + (.P2+P4f) , (35) 

which concludes the proof. ■ 

Lemma 6. For a binary input quaternary output channel with synchronization errors, for any input distribution, 
we have 

i7(r(^)|X) < H{Y\X) + Em {M] H{pi,p2,P3,P4). (36) 

Proof: Substituting the straightforward result H{Y \Yj) = H{pi,p2,P3,PA) in the result of Lemma |2] 
concludes the proof. ■ 

We can now complete the proof of Theorem |2] 
Proof of Theorem |2] : Using the results of Lemmas |5] and |6l we obtain 

/(X;1^W) > I{X;Y)-nr [H{pi,p2,P3,Pi) +log{{pi + Psf + {P2 + P^f)] • (37) 

Hence, due the result in Proposition [H the proof is complete. ■ 
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TABLE III 
TRANSITION PROBABILITIES FOR A BINARY INPUT 5-ARY OUTPUT CHANNEL. 





p{yI'\y,) 


Y^ 


y/"^ = -2 


.i 


-1 


y/^^ = 


y}"^ = 1 


Yj'^^ = 2 


-1 


P2 


Pi 


Po 


P-i 


P-2 


1 


P~2 


P-i 


Po 


Pi 


P2 



TABLE IV 

Transition probabilities for a binary input symmetric 6-ary output channel. 





PiY^'^lY,) 


yj 


y}'^ = -3 


YJ'^ = -2 


Yl^^ = -1 


Yj'^ = 1 


y/^^ = 2 


Y^"^ = 3 


-1 


P3 


P2 


Pi 


P-i 


P^2 


P-3 


1 


P~3 


P^2 


P^l 


Pi 


P2 


P3 



C. Binary Input Symmetric q-ary Output Channel with Synchronization Errors (Odd q Case) 

In this subsection, we consider a binary input symmetric g-ary output channel with synchronization errors for 
an arbitrary odd value of q, where we represent the transition probability values P [Y = k\Yj = bj for different 
values of 6 G {-1, 1} and k = {-^, • • • , -1,0, 1, • • • , ^} by P (yJ'^^ = k\Yj = b\ = p^xb- For instance, 
Table |lll] shows transition probabilities for a binary input 5-ary output channel. 

The main result on the BSQC channel with synchronization errors with odd g is a generalized version of the 
result in Theorem [T] 

Theorem 3. The capacity of the BSQC channel with synchronization errors Cqs for an odd q can be lower bounded 
by 



Cqs >Cs-r[ H{p_3^,- ■ ■ ,Pi^) + log 2pl + ^(pfc +p-k) 



k=l 




(38) 



where Cg denotes the capacity of the binary input synchronization error channel. 

Proof: The proof of the theorem is given in Appendix |A] ■ 

D. Binary Input Symmetric q-ary Output Channel with Synchronization Errors (Even q Case) 

We now consider the generalization of the result of Theorem |2] for even q. For the transition probabilities of 
the binary input g-ary output channel, we define Ply. = k\Yj = h\ = Pkxb< where b G { — 1,1} and k = 
{— !,••• ,— 1,1,--- ,|}. For instance, Table JV] shows transition probabilities for a binary input 6-ary output 
channel. 

The main result on the BSQC channel with synchronization errors for any q is given in the following theorem. 



Theorem 4. Capacity of the BSQC channel with synchronization errors Cqs, for any even q can be lower bounded 
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by 



Cqs >Cs-r 



Hip-i,--- ,P-i,Pi,--- ,Pf) + log "^(pk+P-k) 



(39) 



^k=l 
where Cg denotes the capacity of the binary input synchronization error channel. 

Proof: The proof of Theorem 5] is given in Appendix |B] ■ 

V. Achievable Rates over BI-AWGN Channels with Synchronization Errors 

In this section, a binary synchronization error channel in the presence of AWGN is considered as defined in 
Section III-BI We present two different lower bounds on the capacity of the AWGN/synch channel. 

Before giving the main results on AWGN/synch channel, we would like to make some comments on its 
information stability. 

A. Information Stability of Memoryless Discrete Input Continuous Output Channels with Synchronization Errors 



It is shown in 111511 that the Shannon's theorem holds in any information stable channel. In lUj], the information 
stability of the memoryless discrete input discrete output channels with synchronization errors is proved which 
shows that the Shannon's theorem holds in such a channel. It can be observed that the proofs used in [ij] can be 
also generalized to the continuous output case as discussed in this section. 

To prove the information stability, it is sufficient to prove the existence of the limit 

C= lim —Cn= lim — max/fX:^), (40) 

which is the information capacity of the channel, and the existence of an information stable sequence of two random 
variables iX,Y\, which achieves the capacity of the channel. 

The only difference between the channel considered here with the channel considered by Dobrushin in [1], is that 
in the continuous output case the output symbols belong to an infinite set. However, this difference does not have 
any effect on the steps of proofs. The existence of the limit in Uj, Section IV] is proved based on the memoryless 
property of the channel which also holds in the continuous output case. 

In the case of the existence of an information stable sequence achieving the capacity (111 Section V]), there is 
no need to condition on the discrete output symbol values, and all the reasoning hold for the continuous output 
case as well. The key point in the proof is that the channel is stationary which also holds for the continuous output 
case, such that the same genie-aided channel as the one considered for the discrete output channel can also be 
considered for our case. The genie-aided channel is obtained by inserting markers through the transmission after 
transmitting each block of length k, where the entire length of transmission i^ K = gk + 1 {I < k). 
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The other point in the proof is the number of possibilities in converting the output of the original channel Y 

into the output of the genie-aided channel Y , i.e., |/~^(y)| where Y = fiY'). Since for the continuous output 

max^|/-i(y)| 
case we still have lim > 0, the proof holds. 

g-5-oo g 

Since, both capacity convergence and existence of an information stable sequence which achieves the capacity 
remain valid in the continuous output case as well, we can conclude that the memoryless discrete input continuous 
output channels with synchronization errors are also information stable and, as a result, the Shannon's theorem 
applies in such a channel. 

B. Capacity Lower Bounds for AWGN/Synch Channels 

Here, we present two results on the capacity of an AWGN/synch channel. Both results are generalizations of 
results for the discrete output channels when the number of quantization levels goes to infinity. The first result is 
obtained by employing a uniform quantizer while in deriving the second result a non-uniform quantizer is employed 
which provides a tighter lower bound. 

Theorem 5. Capacity of the AWGN/synch channel Cas can be lower bounded by 

■ — 71) 



CAs>Cs-r\og{J-{l + e-^^y (41) 

where Cg denotes the capacity of the synchronization error channel. 

We give an outline of the proof and defer its details to Appendix O We consider a quantized version of the 
output symbols via a 2M-levels uniform quantizer by quantization intervals of length A with M going to infinity 
and A going to zero. Therefore, for pm (m = {—M, ••• ,— 1,1,--- , M}) which denotes the probability that the 
continuous output symbol, Yj, being quantized to the 6m-th quantization level (6 € {—1, 1}) conditioned on Xj = b 
being transmitted, we obtain 



Pn 






where Q{.) is the right tail probability of the standard normal distribution. By substituting (l42l) into the result of 
Theorem H] we can write 

/ M ^ 



Cas >Cs-r lim 

A/-!>oo,A-s>0 



H{p_M,--- ,P-l,Pl,--- ,Pm) +log ^ {Pm+P- 



■m) 



\m=l 



(43) 



Finally, by using the fact that when M — )• oo and A — )• 0, we have p^ — /(I — ?n.A)A, with f{x) = J- e~^^ , 
after some algebra (detailed in Appendix O. we obtain the result in (|4T]| . 

This result is obtained as a straightforward generalization of the discrete output channel results by employing a 



18 




Fig. 4. Symmetric non-uniform quantizer step sizes. 



symmetric uniform quantizer, but the result may not be tight. For instance, for a = 0, i.e. the noiseless scenario, 
the result does not match with the trivial result which is Cas = Cg for a = 0. We expect that if we apply an 
appropriate non-uniform quantizer on the output symbols of the AWGN/synch channel, we can achieve a tighter 
lower bound on its capacity (which also agrees with the trivial result for Cas = Cs for a = 0). By using this 
idea, we present our main result on the capacity of an AWGN/synch channel in the following theorem by using a 
symmetric non-uniform quantizer. 

Theorem 6. Let Cs denote the capacity of the synchronization error channel, then for the capacity of the AWGN/synch 
channel Cas, we obtain 



Cas d. Cg 



log(e) 



L^-^->mMM^-Mi 



(44) 



Proof: To prove the theorem, we first define an appropriate symmetric non-uniform quantizer with 2M 
quantization levels. Then, by letting M go to infinity and employing the result of Theorem 51 we complete the 
proof. 

In general, by utilizing any symmetric quantizer with 2M quantization levels on the output symbols Yj, for the 
transition probabilities of the resulting binary input symmetric 2M-ary output channel, we have 



Pm = P(y(2M) ^ ^^1^. ^ ^^ 



P{tm-l < Yj < t 



J ^ '"mj 



<m<M 
-M < m < 



(45) 



J~^\ t^m ^ J^ j ^ '-m— ij 

where t_m, = —tm, to = and tm-i < tm for m = {I,- ■ ■ ,M}. We choose the quantization step sizes, i.e., 
^m = tm — tm^i for m = {1, • • • , M}, to Satisfy pi = P2 = ■ ■ ■ = Pm- Note that due to symmetry of the quantizer 
A_m = ^m (as illustrated in Fig. H]). On the other hand, by defining P = Q(^), we have >^ p_m = P and 



M 



m=l 



y Pm = 1 — P which results in pm = ^-jf- for m = {1, • • • , M}. 

m=l 

Using the result of Theorem |4] to derive a lower bound on the capacity of the channel with 2M-level quantized 
outputs, we need to obtain H{p_M, ■ ■ ■ ,P-i,Pi, ■ ■ ■ ,Pm) + log [Y.m=iiPm +P-m)'^)- In the following, we first 
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compute the exact values of Hm = H{p_M, • • • ,P-i,Pi, ■ ■ ■ ,Pm) - log(M) and log [Y.m=iiPm + P-mfj + 
log(M). For Hm, we have 



M M 



Hm = - ^ Pm log(Pm) - ^ P-m \og{p-m) - log(M) 
m=l m=l 

M 

= -{l-P) log(l -P)-Y, P-n. log{Mp.m)- (46) 



m=l 



To calculate — X]m=iP-m^os(^P-'")' ^^ ^^^^ derive a relation between pm and p_m by using the fact that 
Am = A_m- For large M and m = {1, • • • , M}, we have pm = /(I — tm,)Am and p^m — /(I + im)Am, where 
/(x) = J- e~2^. Furthermore, since pm = ^-jj- for m = {1, • • • , M} and {L^"! = e^^, we can write 



r^u 



1 - P 2t„ 

with the understanding that the approximation becomes exact as M — )• oo. By using this result, we obtain 

M MM 

M^oo 



lim - V" p_m log{Mp_rn) = Hm - V" p-rn log (1 - P) - lim V" p_rn log ( 6 ^ 

m=l m=l 

Af 

Plog(l-P)- lim Vp.mlogfe-'^), (48) 



M-5-00 

m=l 



where we used the fact that X]m=i ^-™ ~ -^- ^^ ^^^ other hand, for lim — VJ p_m log ( e ^ ) , we can write 



m=l 
M M 



lim -V'p.mlogfe ^) = lim log(e) V" /(I + tm)Am— 

m=l m,= l 

= log(e)/ /(1 + t)— dt 



'0 

a^ Jo ^/2^Ta 

, X / 2 1 2 \ 

log e ^=^e-^ -—Pj- (49) 



By substituting (|49ll and ([481) into (|46ll, we obtain 



lim Hm = - log(l -P) + log(e) f-J^g-i^ " 4^ ) • (50) 
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At this point, we only need to obtain the exact value of X]m=i(^m +P-m)^» where we have 

M M 



1=1 

M 

1 _ P)2 + 2P(1 - P) + 5] M/_^. (51) 



m=l m=l 

M 



m=l 

Furthermore, if we let M go to infinity, for X]m=i ^P-rw we can write 

lim J]Mp2_^= lim ^M/(l + t^)A^ ;/ "^ p„ 

Af — 5-00 -^ ' M— ^CXD -^ ' f i — tm 

m=l m=l •' ^ ™^ 

M 



lim (1 - P) V ^^e^ 

771=1 



(tm + 1)^ _2tjjj_ 

M- 



(1-P)/ e ^^^e'^dt 

Jo v2vr(T 

(1-P)/ e-^-^^dt 

Jo \/2'ira 

(l-P)e^Qf-V (52) 



Using the results of (152]) and (1501) . we obtain 



lim ( H{p^M, ■ ■ ■ ,P-i,Pi, ■ ■ ■ ,Pm) + log V {pm+P-mf ] = lim Hm + log V M{pm+P-mf 



Finally, by substituting this result into (|39l ). the proof follows. ■ 

By employing a symmetric non-uniform quantizer, we achieve a tighter lower bound on the capacity of the 
AWGN/synch channel compared to the lower bound in Theorem |5] The result is also in agreement with the trivial 
result Cas = Cs (cr = 0). A primary advantage of the derived lower bound in (|44l is that we can use any lower 
bound on the capacity of the synchronization error only channel to lower bound the capacity of the AWGN/synch 
channel. 

VI. Numerical Examples 

In this section, we give several numerical examples of the lower bounds on the capacity of the ins/del/sub and 
del/AWGN channel and compare them with the existing ones in the literature. To the best of our knowledge, there 
are no existing results on lower bounding the capacity of the ins/del/sub/ers and ins/del/ AWGN channels, therefore, 
our results will provide a benchmark for these general cases . 
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A. Insertion/Deletion/Substitution Channel 

In Table |Vl we compare the lower bound on the capacity of the ins/del/sub channel (l20l) with the existing lower 



bounds in \1, 



nil for several values of p^i, Pi and ps- We employ the lower bound derived in [6] as the lower 



bound on the capacity of the deletion channel and the lower bound in lllltl as the lower bound on the capacity of 

n 

the ins/del channel in ( |20b . Note that the Gallager's model in |2] by parameters pd, Pi and pc can be considered as 
concatenation of an ins/del channel with parameters pd and pi, and a BSC channel with cross error probability of 
Ps where ps is the solution of pc = (1 — Ps){^ — Pd — Pi)- The advantage of the lower bound (|20] | is in using the 
tightest lower bound on the capacity of the ins/del channel in lower bounding the capacity of the overall channel, 
i.e., the information rate of the overall channel is lower bounded for the input distribution which results in the 
tightest lower bound on the capacity of the ins/del channel. We observe that for pi = 0, a fixed pd and small values 
of Ps, the lower bound (|20] | improves the lower bound given in 111 ill . This is not unexpected, because for small 
values of ps the input distribution achieving the capacity of the i.i.d. deletion channel is not far from the optimal 
input distribution of the del/sub channel. We also observe that the lower bound (|20l ) outperforms the lower bound 



given in ||2|]. However, for the case pj 7^ it does not improve the lower bound given in llllll . since as the lower 

y 



bound on the capacity of ins/del channel we used the result in 
bound on the capacity of the overall channel. 



and lower bounded further to achieve lower 



TABLE V 

Comparing the lower bound derived on the capacity of the ins/del/sub channel with existing lower and upper 

BOUNDS (Boldface numbers show the best bounds). 



Pd 


Pi 


Ps 


LB from [2J 


LB ^ 


LB from [IT] 


UB from [llj 


0.001 


0.00 


0.001 


0.9772 


0.9775 


0.9773 


0.9856 


0.001 


0.00 


0.01 


0.9079 


0.9082 


0.9081 


0.9163 


0.001 


0.00 


0.1 


0.5201 


0.5204 


0.5210 


0.5292 


0.01 


0.00 


0.001 


0.9079 


0.9107 


0.9091 


0.9586 


0.01 


0.00 


0.01 


0.839 


0.842 


0.842 


0.886 


0.01 


0.00 


0.10 


0.454 


0.458 


0.466 


0.510 


0.10 


0.000 


0.001 


0.5207 


0.5514 


0.5346 


0.7300 


0.10 


0.00 


0.01 


0.458 


0.489 


0.492 


0.644 


0.10 


0.00 


0.10 


0.108 


0.140 


0.211 


0.363 


0.10 


0.10 


0.001 


0.0689 


0.1678 


0.1761 


0.4504 


0.10 


0.10 


0.01 


0.013 


0.0984 


0.139 


0.438 



B. Insertion/Deletion/AWGN Channel 

We now give several numerical examples of the lower bound (|44]) on the capacity of the ins/del/ AWGN channel 
and compare them with existing results. In the literature, there are only a few results on the capacity of the 
deletion/AWGN channel, e.g., the simulation based bound of |il3i] which is the achievable information rate of the 
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Deletion-AWGN Channel 



O 0.: 




■Pd = 0.01 (LB from [13]) 
Pi = 0.01 LB (47) 
■Pd = 0.05 (LB from [13]) 
H- Pd = 0.05 LB (47) 
->— Pd = 0.1 (LB from [13|) 
>- Pi = 0.1 LB (47) 



Fig. 5. Comparison between the lower bound ( I44t with the lower bound in lUBIl versus SNR for different deletion probabilities. 



deletion/AWGN channel for i.u.d. input sequences obtained by Monte-Carlo simulations and the analytical result 



given in jlOll which is a lower bound on the information rate for i.u.d. input sequences, and no previous results are 
available for the ins/del/ AWGN case. 

Fig. |5] shows a comparison of the lower bound on the capacity of the del/AWGN channel in (l44l ) with the results 



in il3ll . We observe from Fig. |5]that the lower bound (04) is far away from the simulation based results of lll3ll for 
large a"^ values and small deletion probabilities. This is not unexpected, because in [13], the achievable information 
rate for i.u.d. input sequences are obtained (through lengthy Monte-Carlo simulations) and i.u.d. inputs are close 
to optimal. However, the procedure employed in [13] is only useful for computing capacity lower bounds for small 
values of deletion probabilities, e.g. p^ < 0.1, while the lower bound in (|29l ) holds for the entire range of deletion 
probabilities by employing any lower bound on the capacity of the deletion channel in lower bounding the capacity 
of the deletion/AWGN channel. We also observe that, since in deriving the lower bound (|44] | on the capacity of the 
deletion/AWGN channel, we employ the tightest lower bound presented on the capacity of the deletion channel, 
for small values of o"^, the lower bound (l44l ) improves the lower bound given in lll3ll . 



VII. Summary And Conclusions 

In this paper, we presented several lower bounds on the capacity of binary input symmetric output channels with 
synchronization errors in addition to substitutions, erasures or AWGN. We showed that the capacity of any channel 
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with synchronization errors which can be considered as a cascade of two channels (where only the first one suffers 
from synchronization errors and the second one is a memory less channel) can be lower bounded in terms of the 
capacity of the first channel and the parameters of the second channel. We considered two classes of channels: 
binary input symmetric q-ary output channels (e.g., for g = 3 a binary input channel with substitutions and erasures) 
with synchronization errors and BI-AWGN channels with synchronization errors. We gave the first lower bound on 
the capacity of substitution/erasure channel with synchronization errors and the first analytical result on the capacity 
of BI-AWGN channel with synchronization errors. We also demonstrated that the lower bounds developed on the 
capacity of the del/AWGN channel for small a"^ values and the del/sub channel for small values of ps improve the 
existing results. 

Appendix A 
Proof of Theorem [3] 

We first give a lower bound on the output entropy of the binary input q-ary output channel with synchronization 
errors related to the output entropy of the binary synchronization error channel, then give an upper bound on 
the conditional output entropy of the binary input g-ary output channel with synchronization errors related to the 
conditional output entropy of the binary synchronization error channel. 

Lemma 7. For a binary input q-ary output channel with synchronization errors, for any input distribution and any 
odd q, we have 

H{Y^'i^) > H{Y) - E{M] log 2pl + Y,{pk+P-kf , (54) 

where Y denotes the output sequence of the synchronization error channel and input sequence of the binary input 
symmetric q-ary output channel, and Y^'^' denotes the output sequence of the binary input symmetric q-ary output 
channel. 

Proof: For p{y^'^'\y-, M = m), we have 

2 

p(y('?)|y,M = m)= [] p^ (55) 

where j^ denotes the number of transitions & — )" f - E.g., in a binary input 5-ary output channel we have p(— 1102|1111) 
P-iPiPoP2- Therefore, for a fixed output sequence y^'^^ of length m with jk symbols of k, since there are 
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g-l 

2""' rifeii (]'')(]'') possibilities for y such that d{y,y^'^^)b^o = jq and d{y,y^'^^)f^^k = i^, we can write 

y,p{ym q=iik=o ^ ''^ i-,=o ^ ''^ 

5-1 
2 

k=l 

= 2-''p™«n(pfc+p_fc)™'^'(^<"), (56) 

fc=l 

where mk{y^'^^) = #{t < m\y^' G {k, —k}}, i.e., the number of the times y/"^ = /c or Y^' = —k. Hence, 



g-i 



2pg + ^(Pfc+P-fc)M . (57) 

By substituting the result of (|57] ) in the result of Lemma [T] we obtain 

H{Y^'^^) > H{Y)-E{M} log Upl + J^iPk+P^kf 

\ k=l 

= -E{M} log I 2pl + Y,(Pk +P-kf , (58) 

which concludes the proof. ■ 

Lemma 8. For a binary input q-ary output channel with synchronization errors, for any odd q and any input 
distribution, we have 

H{Y^'i'^\X) < H{Y\X) + E{M}H{p_^,- ■ ■ ,P^)- (59) 

2 2 

Proof: By using the result of Lemma |2l we can write 

H{Y'^''^\X) < E{M}H{y}''^\Yj) + H{Y\X) 

= E{M}H{p_^,--- ,p^) + H{Y\X). (60) 

2 2 

■ 
Obviously, by employing the results of Lemmas [7] and [8] and using the same approach as in the proof of Theorem[T] 
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the proof of Theorem |3] is complete. 

Appendix B 
Proof of Theorem |4] 

We need the following two lemmas to proof Theorem |4] In the first one, a lower bound on the output entropy of 
the binary input g-ary output channel with synchronization errors is derived relating with the output entropy of the 
binary synchronization error channel. In the second one, we give an upper bound on the conditional output entropy 
of the binary input q-ary output channel with synchronization errors related to the conditional output entropy of 
the binary synchronization error channel. By employing the result of two following lemmas and using the same 
approach as in the proof of Theorem |2] Theorem 5] is proved. 

Lemma 9. For a binary input q-ary output channel with synchronization errors, for any input distribution and any 
even q, we have 

HiY^i^) > H{Y) - E{m} log j ^ (Pfc + P~k? j , (61) 

where Y denotes the output sequence of the synchronization error channel and input sequence of the binary input 
symmetric q-ary output channel, and Y^'^' denotes the output sequence of the binary input q-ary output channel. 

Proof: Due to the result of Lemma [H we have 

H{Y^'i^)-H{Y)>-E^\\og{Y^ Y^ p{y'^'i\y,M = m)p{y^'i\M = m)\\ . (62) 

On the other hand for p{y^''^\y, M = m), we have 

g 
2 

p(y («) I y , M = m) = J] pi'pt^ , (63) 

1 

where jk denotes the number of transitions 6 — > |. For instance, in a binary input 6-ary output channel we have 

p(— 11— 32| 1111) = p-iPiP-3P2- On the other hand, for a fixed output sequence y^^^ of length m with j^ symbols of 

k, there are 11^=1 H^) Hz'') possibihties for y such that d{y, y^''^)fy^k = i^.. By defining nikiy^''^) = #{t < rn\y^ G {k, —k}} 
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i.e., the number of the times Y^ = k or Y^ = — A;, we can write 



= iiiPk+p-kT^'-' 



k=l 



l[{Pk+p-kr^^y'''\ (64) 



k=l 



Furthermore, by taking the summation over all the possibilities of y^^^ in (l64l ). we obtain 

2 



■+mq=m \ -■-' ' 2^ «=1 fc=l 

\ m, 

J]bfc+P-fc)M ■ (65) 



mi-\ hmq =m 



,fc=l 

By substituting the result of (165] ) in (I62l ). we obtain 

i/(y('')) - /f (Y) > - log j YiPk + P-k? I E "^^("^) 

\fc=l y m 

= -EjM} log ^(pfc + p.k? , (66) 

which concludes the proof. ■ 

Lemma 10. In any binary input q-ary output channel with synchronization errors, for any input distribution and 
any even q, we have 

H{Y^''^\X) < H{Y\X) + E{M}H{p_,,- ■ ■ ,p-i,pi, ■ ■ ■ ,P|). (67) 

Proof: The proof is similar to the proof of Lemma |8] ■ 

Appendix C 
Proof of Theorem [5] 

We first compute if a = H{p_M, ■■■ ,P-i,Pi,--- ,Pm) + log(A) and Y.m=i S (Pm + P-mf for M -> oo and 
A — )• 0. Then by employing the result of Theorem lU we prove the theorem. 

For large M, we have p^ = /(I — m,A)A with the understanding that the approximation becomes exact as 
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A — ^ where f{x) = J- e~2^. Therefore, for H/^ = H{p_M, ■ ■ ■ ,P-i,Pi, ■ ■ ■ ,Pm) + log(A), we can write 



27ro- 



M r 



lim H/\ = lim — } 



M->oo,A-s>0 



Af^oo,A-s>0 



m=l 



/(I - mA) log(/(l - mA)) + /(I + mA) log(/(l + mA)) 



/(I - x) (^log{V2^a) + ii^ log(e)) + /(I + x) (^log(^/2^a) + ^^^^ log(e) 

/(I - x) (\og{V2^a) + ^^~J^ logle)") dx 
log(e) 



dx 



log(V27ro-) + 



(68) 



^M 1 



On the other hand, for X]m=i S (Pm + P-m) , by letting M — >• cx) and A — )• 0, we obtain 



M ^ 

.^^ A^n ^ — ' A 



M-5-oo,A-5>0 ^-^ A 
m=l 



Af 

p-mf= lim V(/(l-mA) + /(l + mA))'A 

A4'->oo,A— >0 -^ — ' 
m=l 
/■oo 

= / {f(l-x) + f{l + x))'dx 
Jo 

^ / ( /( V2(l - x)) + /(^/2(1 + x)) + e' 
ina Jo ^ 

(1 + e-^). 



27rc7 
1 



dx 



(69) 



Using the results of (l68T l and (|69] |. we can write 



.r ^™A J ^(P-M,--- ,P-l,Pl,--- ,PA/) 

Af-^>oo,A-s>0 



/ M \ 

+ log I ^(Pm+P-»n)^ 1 
\m=l J 

\^^ I -H'a + log I ^ -^{Pra+P-mY 
^ \ \m=l 



lim 

A/->oo 



log (a/ 2(1 + ^" 



(70) 



Finally, by substituting this result into (1391 ). the proof follows. 
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